Skip to content

Sync upstream transformers main into OMY - #10

Merged
Qubitium merged 7 commits into
mainfrom
sync-upstream-main
Aug 28, 2026
Merged

Qubitium merged 7 commits into
mainfrom
sync-upstream-main

Conversation

@Qubitium

Copy link
Copy Markdown

What does this PR do?

Syncs the latest huggingface/transformers main into OMY and resolves the merge conflict in src/transformers/modeling_layers.py.

Changes

  • Merged upstream/main into OMY main.
  • Resolved the conflict in src/transformers/modeling_layers.py by keeping import pcre as re and adding the new import sys from upstream.
  • Updated src/transformers/models/vibevoice/convert_vibevoice_to_hf.py to use import pcre as re so OMY's regex standardization stays consistent after the sync.

make style passes locally.

Link to Devin session: https://app.devin.ai/sessions/92d48d7ab1a64fc98c92fc894cd1dadb
Open in Devin Desktop: https://app.devin.ai/desktop/session/92d48d7ab1a64fc98c92fc894cd1dadb?variant=devin
Requested by: @Qubitium

ydshieh and others added 7 commits August 27, 2026 14:31
* [debug] Poll GitHub API 10x to observe cache inconsistency in schedule runs

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* Retry get_daily_ci_runs when GitHub API returns stale cache results

The GitHub Actions search index (used for event=/branch= filters) can lag
behind the database — different backend nodes return wildly different
total_count values (190, 238, 311, 413 observed for the same URL within
minutes), and the most-recent runs are missing from stale responses.

Detect staleness by checking whether the current GITHUB_RUN_ID appears in
the returned list; if absent, retry up to 3 times with a 30s delay.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* [debug] Use get_daily_ci_runs in debug job; add current_run_id param

Add optional `current_run_id` parameter to `get_daily_ci_runs` so callers
can supply a known schedule run ID for stale-cache detection when
GITHUB_RUN_ID belongs to a non-schedule-triggered job (e.g. push).

Update the debug workflow job to call the Python function directly
(two quick back-to-back calls) instead of raw curl, so the retry logic
is exercised end-to-end and visible in the CI logs.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* [debug] Force retry path with fake run ID to verify stale-detection logging

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* Revert "[debug] Force retry path with fake run ID to verify stale-detection logging"

This reverts commit e49340f.

* [debug] Run get_daily_ci_runs 20 times to catch stale cache hit

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* [debug] Sleep 60s between get_daily_ci_runs calls to hit different cache nodes

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* Increase get_daily_ci_runs max_attempts from 3 to 5

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* Add unit tests for get_daily_ci_runs stale-cache retry logic

- tests/utils/test_get_previous_daily_ci.py: 5 tests covering fresh hit
  (no retry), stale→fresh (one retry), all stale (max_attempts exhausted),
  no current_run_id (stale check skipped), and empty schedule fallback to
  workflow_run event
- utils/get_previous_daily_ci.py: ruff formatting fix (long print line)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* Infer stale-check eligibility from current run's workflow_id

Instead of an explicit `current_run_id` argument, `get_daily_ci_runs` now
always fetches the current run's metadata (GITHUB_RUN_ID) upfront and
compares its `workflow_id` to the queried one:

- Same workflow → stale-cache detection applies (max_attempts=5, 30s sleep)
- Different workflow → skip (e.g. AMD CI querying Nvidia CI runs); left for
  a follow-up PR once the same-workflow case is confirmed stable

Also adds a TODO comment on the event=workflow_run fallback path (AMD CI)
and updates the 6 unit tests accordingly (new test for the different-workflow
case; all tests account for the upfront current-run lookup call).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* Gate stale-cache check on GITHUB_EVENT_NAME == 'schedule'

Push- and dispatch-triggered runs won't appear in event=schedule results
even when the API is fresh, causing spurious retries. Only enable the
stale-cache retry loop when the current run is itself schedule-triggered.

Adds a corresponding unit test (test_non_schedule_event_skips_stale_check).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* Revert debug changes to self-scheduled-caller.yml

Restore the workflow to its main-branch state; the debug polling job was
only needed to verify the stale-cache behaviour.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* Condense get_daily_ci_runs comments and refer to huggingface#48374 for rationale

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

---------

Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
…ckend (huggingface#48341)

* Fix `safe_open` mmap memory exhaustion on Windows by using `pread` backend

On Windows, memory-mapping safetensors files reserves copy-on-write commit
charge for the entire file. For large multi-shard checkpoints this exhausts
virtual memory. Switch to the `pread` backend on `win32` in both
`_load_pretrained_model` and `MtpModel` loading paths.

* Address review: add MPS backend fix to MtpModel, hoist backend selection before loop, use targeted sys.platform patch in test

* Address review: remove tests
…uggingface#48191)

* [ONNX] Skip affected models on torch >= 2.13 (two dynamo ONNX regressions)

torch 2.13.0 introduced two regressions in dynamo ONNX export:
- pytorch/pytorch#194381: aten.sub type-promotion failure for scalar - int_tensor
- pytorch/pytorch#194382: aten.mul.Scalar missing ONNX decomposition

Skip all 35 affected model classes under `EXPORT_SKIPS["onnx"]` when
torch >= 2.13, using the existing skip infrastructure. Will be removed
once the upstream PyTorch fixes land.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* Scope ONNX skip to torch == 2.13.x only (auto-runs on 2.14+)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* Add missing BigBird subclasses to torch 2.13 ONNX skip list

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* [ONNX] Clarify skip guard is torch == 2.13.x only (not >=)

Update comment and skip reason strings to say `torch == 2.13` instead of
`torch >= 2.13`, making it clear the guard auto-lifts on 2.14+.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* [ONNX] Fix the torch 2.13 scalar regressions instead of skipping the models

Both regressions are a Python float meeting an integral tensor, whose promotion
torch 2.13 mishandles: `1.0 - int_mask` crashes the decomposition pass
(pytorch/pytorch#194381) and `int_mask * 2.0` reaches translation with no
registered ONNX decomposition (pytorch/pytorch#194382). On the same torch,
`1.0 - float_tensor` and `float_tensor * 2.0` export fine — so promoting the
tensor operand up front, to the dtype the op already produces, is enough. The
op and its overload are left alone, and the inserted cast carries the op's own
`meta` because for these elementwise cases it is the same value.

Decomposition also emits `mul.Scalar` with a *symbolic* second operand (a
division result, not a literal). There is no constant to promote there, so that
one is rewritten to `mul.Tensor`, which has the two-operand translation — the
same rewrite `_fix_remainder_scalar` makes for the same reason. Reachable
because the FX fixes run again right after `run_decompositions`.

This drops the 39 skip entries: the 15 affected families export again on 2.13,
including the vision models (Sam, SamHQ, GotOcr2, GroundingDino, SegGpt,
EfficientLoFTR, DeepseekOcr2).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* Resolve the promoted-op set on first use, not at import

`exporter_onnx` is importable without torch — the CI job that imports transformers
with PIL only proved this the hard way — and naming `torch.ops.aten.*` overloads in
a module-level frozenset broke that with a `NameError` before anything ran.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* Say what the `mul.Scalar` operand is, and check it

Review raised two things. The `Node` check assumed a tensor operand: it is never one. Across the affected
families all 33 sites are an `operator.truediv` result, i.e. a `SymFloat`, which `mul.Tensor`'s translation
does take — so the rewrite stands, but the guard now names the forms that op accepts instead of trusting
the node type, and anything else keeps the `mul.Scalar` overload and fails visibly in translation.

The fixes are also deliberately not version-gated, which the docstring implied they were. Both rewrites
are semantics-preserving on any torch — a cast to the dtype the op already produces, and an overload swap
with the same meaning — so gating would only decide which torch exercises the path.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

---------

Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: IlyasMoutawwakil <moutawwakil.ilyas.tsi@gmail.com>
Co-authored-by: Ilyas Moutawwakil <57442720+IlyasMoutawwakil@users.noreply.github.com>
…d output (value drift) (huggingface#48376)

Update Qwen3VLMoe batch integration test expected output (value drift)

Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
…to avoid MoE disk offload issue (huggingface#48377)

* [LongcatFlash] Fix test_longcat_generation_cpu by using device_map="cpu"

`device_map="auto"` causes accelerate to offload MoE expert weights to disk,
which then fails to reload them due to an internal weight format incompatibility.
Since the test already requires large CPU RAM, use `device_map="cpu"` to keep
all weights in memory and avoid disk offloading entirely.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* [LongcatFlash] Update golden string and skip test_longcat_generation_cpu on small runners

- `test_shortcat_generation`: update expected output to current model output (value drift)
- `test_longcat_generation_cpu`: replace `@require_large_cpu_ram` with
  `@require_torch_accelerator_memory(memory=1100)` — the 562B parameter model requires
  ~1,047 GiB of bfloat16 weights, far exceeding the CI runner budget (84 GiB single /
  168 GiB dual), and disk offloading fails due to MoE weight format incompatibility
  with accelerate

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* remove unused require_large_cpu_ram import

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

---------

Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
@devin-ai-integration

Copy link
Copy Markdown

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

  • Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
  • Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

  • Disable automatic comment, CI, and merge conflict monitoring

@Qubitium
Qubitium merged commit dec3721 into main Aug 28, 2026
28 of 32 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants